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PERSONAL VIRTUAL ASSISTANT 

This application is related to PCX Application Serial No. , entifled 

Virtual Assistant Engine, which is filed simultaneously herewith, assigned to a 
5 common assignee, and are hereby incorporated by reference. 

FIELD OF THE INVENTION 
The present invention relates to a computer-based, pCTSonal virtual assistant 
for managing conmiunications and information. 

BACKGROUND OF THE INVENTION 

10 Mobile professionals, such as physicians, attorneys, sales representatives and 

other highly mobile professionals often find it difficult to communicate with clients, 
customers, colleagues and assistants. These mobile professionals travel fi:equently 
and are not accessible via a desk telephone or traditional, wired computer network. 
They typically employ hunan assistants to relay important information, maintain tiheir 

1 5 schedules and filter out all unnecessary interruptions. The virtual assistant of tiie 
present invention allows the mobile professional to access personal, company, and 
public information, including contacts, schedules, and databases firom any intmctive 
device, such as telephone. 

Electronic assistants with voice interfaces are known. U.S. Patent No. 

20 5,653,789 to Miner, et al. discloses a mefliod implemented by a computer-based 
electronic assistant to receive and manage incoming calls to a subscriber. The 
electronic assistant in Miner, however, does not disclose a virtual assistant whose 
underlying behavior can be changed by Ihe user or who has any degree of automatic 
adaptivity. 

1 
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Vojice response systems (VRS) that automatically adapt to the user are known. 
For example, U.S. Patent No. 5,483,608 to O'Sullivan discloses an interactive VRS 
that automatically adapts to suit the speed at which the caller interacts with the 
system. The VRS disclosed in 0' Sullivan is programmed to measure flie response 
5 times of the caller and adjust the playing speed of the application dialogue's voice 
messages accordingly using an algorithm incorporated into the application software of 
the voice response system. Thus, if the caller is responding relatively fast and without 
error to the voice message prompts, the system will gradually speed up subsequent 
voice message prompts. If the caller is responding more slowly to the voice message 
10 prompts or is making errors in their responses, the system will slow down subsequent 
voice message prompts. The system disclosed in O'Sullivan, however, does not 
perform the actions of a virtual assistant, nor does it permit the user to control how the 
system ad^ts. 

Another caller adaptive VRS is disclosed in U.S. Patent No. 5,553,121 to 
1 5 Martin et al. Martin et al. discloses a system for varying the voice menus and 
segments presented to the user of a voice response syst^ according to the 
competence of the user. The response time of a user to voice prompts is measured 
and an average response time is determined. It is assumed that the lower the average 
response time, the greater the competence of the user. The average response time is 
20 used as an index to a table of ranges of response times. Each range has respective 
voice segments associated therewith. The voice segments comprise oral instructions 
or queries for the user and vary according to the anticipated competence of the user. 
If the average response time changes such that the voice segments indexed are 
different to the current voice segments then a data base containing information 



\ 
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relating to user competence is updated to reflect such a change. Accordingly, when 
the user next interacts widi the voice response system a new set of voice segments 
more appropriate to the usefs competence will be played. The system in Martin et al. 
also discloses determining user competence by identifying individual callers using 
5 existing caller identification technology. The call identification code of a telephone 
call can be used as an index to data stored in a user database comprising information 
relating to the competence of a user. Altematively, the user can be asked to enter a 
password before fiirdier access is allowed to the system. The password can &en serve 
as an index to the stored data associated with the user. The stored data identifies 

10 which set of voice data is appropriate for use during an interaction with said user. 

Alternatively, determining the number of times per day that a user accesses the system 
or the length of time which a user has subscribed to such a system may also be 
indicative of then: competence. Again, VRS disclosed in Martin does not perform the 
functions of a virtual assistant, nor does it permit the use to have any significant 

1 5 degree of control over the behavior of the system. 

Further, while the prior art systems adapt automatically to the caller, the 
degree of adaptation is relatively limited. For example, the prior art systems do not 
disclose a virtual assistant that automatically uses words associated with polite 
discourse when the user's input contams words associated with polite discourse. Prior 

20 art systems also do not disclose a virtual assistant that adapts to the user based on the 
user's emotional state. 

Another important function provided by a virtual assistant, which is also used 
ubiquitously by busy professionals who do not have a virtual assistant, is voice mail. 
Conventional voicemail systems, as well as virtual assistants with voicemail 

3 
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functionality, however, have disadvantages that limit fheir usefufaiess. For example, a 
professional, such as a physician or an attorney, is a person that many people attempt 
to contact throughout the day. Such persons, however, cannot for various reasons be 
reached directly by telqjhone. Thus, many such professionals have voicemail. If the 
5 professional is not able to regularly access his or her voicemail, at the end of the day, 
or whenever the voicemail is accessed, the number of voicemail messages may be so 
large so as to be virtual^ unmanageable because the only option is to listen to the 
messages one at time in sequential fashion. This can cause the voicemail user to 
simply abandon use of the voicemail system because the user does not have time to 

1 0 listen to a large number of messages one at a time. This makes it even more difficult 
for persons attempting to contact the busy professional, the only remaining options 
being to contact the professional in person, which is impractical, sending a page, 
sending an email or oth^ know contact metiiods. If the volume of pages or emails 
becomes so great, tiie cycle is repeated. 

1 5 Navigating large databases of non-messaging information, such as a contacts 

database is also quite cumbersome and difficult with conventional voicemail systems 
and virtual assistants, particularly, when the database is large and the user is 
attempting to access the database with a voice user inter&ce. 

U.S. Patent No. 4,488,005 to Frantz discloses a telq)hone answering system 

20 fliat provide some limited ability for the selective retrieval of messaging information, 
but Frantz system has several disadvantages. One disadvantage of Frantz is that Ihe 
user of the answer system is required to manually input tiie word or words that are to 
be used later for selective retrieval of the messages. Moreover, the system disclosed 
in Frantz is limited to retrieval of recorded voice messages. It does not permit 
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selective letrieval of non-messaging infoimation, such as infonnation in a contacts 
database or any other electronically accessible database. 

U.S. Patent No. 5,558,540 to Greco, et al. discloses a graphical user interface 
for the display of voice mail messages and playing voice mail messages tfarougih a 
5 computer speaker. Using a mouse, the messages can be manipulated individually or 
in groups. The user can listen to the messages in any order at any time. A 
disadvantage of the system disclosed in Greco et al. is that the user is required to have 
a personal computer to manipulate the message information. 

U.S. Patent No. 5,469,491 to Morley, Jr., et al, discloses a telephone 
10 answering system that allows selective retrieval of the messages by dialing an 
operator DTMF tone to have the recorded textual messages read by the operator 
and/or activating the playback of the recorded voice messages by inputting 
appropriate DTMF tones. A disadvantage of the system disclosed in Morley, Jr., et al. 
is that Hie user is required to memorize or have access to the proper DTMF tones to 
1 5 selectively retrieve messages. 

A conventional computer {^plication witii a voice usct interface, such as a 
virtual assistant, needs to ascertain user choices at all junctures where user input is 
required. Such conversational dialog systems typically ask questions, such as, 
"Would you like to include a message?" that must be answered by saying *ye^* or 
20 '^o." Alternatively, they provide menus where a niunber of choices are listed for the 
user to select, for example, or "Would you like to send your message, review your 
message, add more to your message or discard your message?" In fhis case, fhe user 
is required to listen to all of the enumerated choices before he or she is able to input 
his or her choice. The choice can be made by speech or by pressing a particular key 
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on a telephone keypad Such systems, however, create artificial interactions, are 
cumbersome and can be annoying to the user, particularly, the skill and/or impatient 
user. 

In addition, while such conventional systems work reasonably well under 
5 good teleconamunication conditions, short utterances, however, such as **y6s" or **no," 
can cause speech recognition errors. This is because the short length of the utterance 
does not always provide sufficient data for accurate speech recognition. Furthermore, 
when signal quality is bad, as can be the case with wireless communications, speech 
recognition errors can reach unacceptable levels. Thus, a reliable method for 
10 ascertaining user choices under such conditions is needed. 

SUMMARY OF INVENTION 
The present invention relates to a personal virtual assistant with many discrete 
features, each of which comprises a separate but related invention. Thus, one aspect 
of the present invention is a computer-based virtual assistant fhe behavior of which 
IS can be changed by the user, comprising a voice user inter&ce for inputting 

information into and receiving information firom the virtual assistant by speech, a 
communications network, a virtual assistant application running on a remote 
computer, the remote computer being electronically coupled to the user interface via 
the communications network, wherein the behavior of the virtual assistant changes 
20 responsive to user input 

Another aspect of the present invention is a computer-based virtual assistant 
that automatically adapts its behavior comprising a voice user interfiice for inputting 
information into and receiving information firom the virtual assistant by speech, a 
communications network, a virtual assistant application running on a remote 
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computer, the remote computer being electronically coupled to the user interface via 
the coromunications network, wherein the remote computer is programmed to 
automatically change the behavior of the virtual assistant responsive to input received 
by the virtual assistant As detailed below, the virtual assistant adapts to the user in 
S many different ways based on the input die virtual assistant receives. Such input 
could be user information, such as information about the user's experience, the time 
between user sessions, the amount of time a user pauses when recording a message, 
the user's emotional state, whether the user uses words associated with polite 
discourse, and the amount of time since a user provided input to the virtual assistant 

10 during a session. 

Another aspect of the present invention is a virtual assistant application with a 
voice user interface that employs the concept of semantically tagging electronic 
information, such as voicemail messages, email messages, contacts, or any other type 
of electronically accessible information. Messages, for example, can be tagged with 

IS information that the user would find helpful in navigating a large set of messages. The 
field to be tagged might be status, that is, whether the message is urgent. Voicemail 
messages also can be tagged based on the sender of the message, which could be 
determined either by automatic caller identification or by the sender stating his or her 
name, the delivery date of the message, or the status of the message, that is, whether it 

20 has be^ read. When messages are marked with this semantic information a virtual 
assistant is able to make use of these tags and present the user with a more organized 
mefliod of accessing his or her messages. 

Another aspect of the present invention is a computer based method for 
perfoiming a first coimnand via a voice user interface on information stored in a 
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computer memoiy, wherein the infonnation upon which the first command is to be 
performed is comprised of a subset of objects specified by the user, the subset being 
selected firom a set of objects, wherein each object in the set of objects has an object 
type, at least one taggable field associated with the object type, and a corresponding 
5 value for the at least one taggable field. The method is comprised of Ifae steps of 
storing the set of objects in the computer memory, receiving firom the user via the 
voice user interface an utterance, &e utterance being comprised of a first command, 
an object type selection, a taggable field selection, and a value for the selected 
taggable field, responsive to the utterance, retrieving a subset of objects from the set 

10 of objects, the subset containing at least one object, the at least one object being of the 
object type selected by the user, having a value in the taggable field selection that 
matches the taggable field value received from the user, performing the first 
command on the retrieved subset of objects performed on an active object in the 
subset of objects, the at least one active object being the object upon which the first 

15 command was performed, receiving a second command from the user, the second 
conomand to be performed on an active object in fte subset of objects, the object 
bemg the object upon which the first command was performed, and performing the 
second command on the active object 

The object type is a voicemail message, email message, meeting request task 

20 request or fax. The taggable field for the voicemail message type is caller's name, 
callback number, delivery date, subject or status. The taggable field for the email 
message type is sender, delivery date, subject or status. The taggable field for the 
meeting request message type is sender, delivery date, subject, location, start time, 
end time or status. The taggable field for the task request message type is sender. 
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deliveiy date, subject, due date, ovmer, status or percentage complete. The taggable 
field for the fax message type is selected from the group consisting of telephone 
number of the machine that sent the fax, sender, delivery date, subject and status. The 
taggable field value for the status field is first, last, new, old, read, unread, deleted or 
5 urgent 

The first command is count, browse, list or read. The second command is 
next, previous, first, last, read, get more detail, file, delete or restore. The object type 
also can be a contact, wherein the taggable field is first name, last name or nickname. 

Another aspect of the present invention is a computer application with a voice 
1 0 user inter&ce, such as a virtual assistant, that provides a user with choice prompts by 
speech and permits the user to select the desired choice by indicating his or her 
selection during the time that the desired choice prompt is being provided by the 
computer application. In one embodiment, the user indicates bis or her desired choice 
by pressing a predetermined key on a telephone keypad, such as the star ("*") key. In 
1 5 anotiier embodiment, the user indicates his or her desired choice by providing verbal 
input to the virtual assistant 

Another aspect of the invention is that the user can selectively determine 
whether to enter a mode of interacting with a computer application with a voice user 
interface whereby with tiie users select a desured choice by indicating his or her 
20 selection during the time fliat the desired choice prompt is being provided by the 
computer application. 

Another aspect of the present invention is a method for receiving &am a 
virtual assistant application user a desured choice fi:om a plurality of choices presented 
to the user by the virtual assistant The method is comprised of receiving a predefined 

9 
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input from the user indicating that the user desires to enter a choice prompt mode; 
providing a choice stream to the user, the choice stream being comprised of at least 
one choice prompt, the choice prompt being associated with a choice available to the 
user and the choice prompt being comprised of a recorded utterance, and a 
5 corresponding active segment, wherein the user can select the choice associated with 
the choice prompt during the corresponding active segment; receiving the predefined 
input firom the user during the active segment corresponding to the desired choice 
prompt, the predefined input tiiat indicating that the user desires to select the choice 
associated with the choice prompt; performing an action associated with the choice 
10 selected by the user, and receiving the predefined input firom the user indicating that 
the user desires tp exit the choice prompt mode. The length of the active segment is 
not less than 200 milliseconds. The input received from flie user during the active 
segment is predefined verbal input, such as the words **yes" and **no." Alternatively, 
the input is predefined nonverbal input, such as a predefined DTMF. The 
15 predetermined DTMF could be the DTMF that is generated by a telephone keypad 
when the star key is pressed. 

The method is fiirther comprised of the step of providing a predefined audible 
signal to the user, the audible signal signifying to the user the beginning of a choice 
stream, prior to the step of providing die choice stream. 
20 The first choice prompt in the choice stream is the choice automatical^ 

determined by the virtual assistant to be the choice likely to be selected by the user 
based on the user's past choices. 

Other features and advantages will become apparent based on the following 
detailed description of flie preferred embodiments and the claims. 

10 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG, 1 is an overview of the virtual assistant (VA) of the present invention; 

FIG. 2 is a diagram of the VA Server, 

FIG. 3 is a diagram of the VA Studio; 
5 FIG. 4 is a diagram oftheVA Engine conceptual model; 

FIG. 5 is a diagram of the VA Manager conceptual model; 

FIG. 6 is a screen shot of the Microsoft Management Console for managing 
flie VA Server Manger, 

FIG. 7 is a screen shot of a web page that uses Active Server Pages to manage 
10 the VA Server Manager, 

FIG. 8 is a diagram of the component relationships of a VA Server Set; 

FIG. 9 is a diagram of a relatively small VA system; 

FIG. 10 is a diagram of a large VA system; 

FIG. 11 is a diagram ofa very large VA system; 
15 FIG. 12 is a diagram ofa hardware configuration for a single ISDN PRI link; 

FIG. 13 is a diagram ofa hardware configuration for a two ISDN PRI links; 

FIG. 14 is a screen shot of the Custom Component Selection screen; 

FIG. 15 is a screen shot of GlobalCall Feature Selection screen; 

FIG. 16 is a screen shot of the Outlook Feature Selection pane; 
20 .FIG. 17 is a screen shot of the VA Management Console; 

FIG 1 8 is of the VA Management Console with the general information form 
displayed in the right panel; 

FIG. 19 is a screen shot of the Add Application Instance Dialog box; 

FIG. 20 is a screen shot of tbe Select TPS Server Dialog box; 
25 FIG. 21 is a screen shot of the Add Recognition Server Dialog box; 

FIG. 22 is a screen shot of the Add VA Engine Dialog box; 

FIG. 23 is a screen shot of the Set Application File Dialog box; 

FIG. 24 is a screen shot of &e Add Process Dialog bo>^ 

FIG. 25 is a screen shot of the Properties Display Panel for a Resource 
30 Manager Service; 

FIG. 26 is a screen shot of the Alert Configuration Interface; 

11 
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FIG. 27 is a screen shot of the TTS Dictionary Display; 
FIG. 28 is a screen shot of the Dictionary Entry Dialog box; 
FIG. 29 is a screen shot of the Database Manager Panel; 
FIG. 30 is a screen shot of &e Mailbox Properties Dialog with Virtual 
5 Assistant Tab; 

FIG. 3 1 is a screen shot of the General tab on the Virtual assistant Preferences 

screen; 

FIG. 32 is a screen shot of the Phone/Pager Tab on flie of the Virtual assistant 
Preferences screen; 

10 FIG. 33 is a screen shot of the VA Interaction tab on the of the Virtual 

assistant Preferences screen; 

FIG. 34 is a flow chart that illustrates a call flow based on different tempo and 
assertiveness settings; 

FIG. 35 is a screen shot of flie Phone Schedule screen; 
1 5 FIG. 36 is a screen shot of the Virtual assistant Tab; and 

FIG. 37 is a diagram of a choice prompt stream with corresponding active 
segments. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The subheadings used herein are meant only so as to aid the reader and are not 
20 meant to be limiting or controlling upon the invention. Generally, the contents of each 
subheading are readily utilized in flie other subheadings. 

Overview 

Mobile professionals, such as physicians, attorneys, sales representatives and 
other highly mobile professionals often find it difiScult to communicate with clients, 

25 customers, colleagues and assistants. These mobile professionals travel firequently 
and are not accessible via a desk telephone or traditional, wired computer network, 
niey typically employ human assistants to relay important information, maintain their 
schedules and filter out all unnecessaiy intemq)tions. The virtual assistant of the 
present invention allows the mobile professional to access personal, company, and 

30 public information, including contacts, schedules, and databases fi:om any interactive 
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device, such as telephone. 

The virtual assistant (**VA") system of the present invention is comprised of 
two main components: (1) the VA Server, which is built on a Windows NT telephony 
server platform, and (2) the VA Studio, which allows skilled information technology 
5 professionals to develop VA applications that inter&ce with electronic messagmg 
systems, such as Microsoft Exchange and Lotus Notes. Hie VA Server is a 
component of the Service Deployment Environment ("SDE"), which is discussed in 
more detail below. The VA Studio is a component of the Service Creation 
Environment ("SCE"), which is also discussed in more detail below. 

10 As shown in Figure 1, the VA Server 10 is comprised of a human intet&ce 12 

and a network inter&ce 14 for handliug calls and providing automated access to 
information to corporate 28, private 30 and pubic 32 information repositories and 
sources. The human interface 12 is comprised of a graphical user inter&ce 22, which 
may be a web browser, a subscriber (or user) voice user interface 24, generally 

1 5 accessed by a telephone, and a public voice user interface 26. The virtual assistant v 
allows a user to use a voice interactive device, such as a telephone, either wired or 
wireless, to access and update such information. The VA Server also manages all 
incoming communications by sorting, prioritizing, and filtering such communications, 
while providing notice to the user of unportant messages and events. 

20 VA Server 

As seen in Figure 2, a core component of the VA Server 40 is the voice- 
enabled Virtual Machine 42, which is also referred to as the VA l^gine. The VA 
Engine receives spoken commands, interprets and executes them. The VA Engine 
supports a COM interface 44, which in turn enables VA applications to provide voice 
25 access to network applications. 

The VA Engine also supports a telephony inter&ce 46 to voice messagmg 52 
and private branch exchange systems S4, enabling third-party systems to be integrated 
with the VA Server. 

The VA Server confomis to Windows NT telephony and speech inter&ce 
30 specifications. The voice messaging interface 56 supports the VPIM (Voice Profile 
for Litemet Mail) standard, and provides a gateway between proprietary voice 

13 
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messaging systems and VA Server. 

The VA system management services provide operations, administration and 
maintenance capability (OA&M) 60. The OA&M applications also provide a Simple 
Network Management Protocol ("SNMP") interface to third parly management 
5 applications, such as HP Openview and CA Unicenter. 

In the preferred embodiment, the VA Server is operable on Windows NT 
Server, release 4.0 or higher, in both single and multiprocessor configurations. Those 
skilled in the art, however, recognize that the VA Server can be ported to other 
computing platforms. Multiple systems may be clustered together to support higher 
10 system workloads and fail-safe operation. 

VA Application 

The VA Application, in the preferred embodiment, is compatible with a 
messaging server 62, such as Microsoft Exchange/Outlook. The VA's architecture, 
however, advantageously permits integration with other commercially available and 
1 5 customized messaging applications. The VA Application can be easily modified to 
satisfy specific requirements of a user. The basic functions of the VA Application 
include: 

Messaging - voice-mail, e-mail, and fixes 

Contact Management. - schedulmg, plamimg, group calendar, contact and 
20 referral organization 

Call Control - remote users to perform conference calling and call 
management; notification and forwarding features allow remote users to be contacted 
immediately by phone^ager when they receive specific voice-mails, e-mails, faxes, or 
pages 

25 Internet Applications - users can access and internet via an internet server 64 

and obtain public information such as weather, travel, financial, competitive data and 
news 

Intranet Applications - users can remotely access information contained on a 
corporate network (inside the company firewall) using the VA, for example, customer 
30 data, shipping and inventory information, sales reports, and financial data, or any 
information on a database server 66, including SQL databases such as Oracle or 

14 
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Infonnix. 

Customer Relationship Management applications - the VA Server integrates 
wifli commercially available customer relationship management (CRM) software 
applications 70, such as Siebel, Pivotal, Sales Logix and Onyx. 

5 VA Studio 

As seen in Figure 3, the VA Studio 80 is comprised of a grammar generator 82 
and a publishing toolkit 84. The VA Studio allows a user to create, modify and debug 
applications that run on the VA Server 40 without requiring the user to be skilled in 
the complexities of the underlying components of the VA Server, such as the speech 

10 recognition engine, text to speech engine, switch control and unified messaging. 

VA Studio employs a graphical user interface (GUI) application that runs on a 
Windows NT workstation. It allows developers to create projects, each of which 
defines a VA application. VA Studio is a multiple document interfece (MDI) 
application that follows the workspace-based model. 

15 The VA Studio follows the Microsoft Component Object Model (COM). VA 

applications are developed using Active Scripting languages such as VBScript and 
JScript, thus enablmg integration with a variety of third party conq)onents. The VA 
applications created with flie VA studio will include voice query to SQL databases, 
message stores, business logic and mainfifame applications. 

20 VA applications are composed of discoiu:ses and resources. Discourses are the 

context of conversations between a user and the VA. Resources are items like voice 
prompts and dictionaries. A developer can utilize the VA Studio Wizard to generate a 
"skeleton" VA application template. Application tenq)lates consist of packages of 
predefined discourses and resources. Discourses are the context of conversations 

25 between a user and the VA. Resources are items like voice prompts and dictionaries. 
Once a VA application template is g^erated, the application is further customized 
using any supported Active Scripting languages. 

After writing the VA plication, it is then submitted to ttie build process. 
During Ihe build process, VA Studio checks for dialog enrors, builds a master 

30 intermediate grammar and builds a master lexicoiL Once compiled and error-fiee the 
application is ready to be published. 

15 
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When an application is published, it is transported from the VA Studio to tiie 
VA Server. The VA Server allows a scripted application to access services such as 
voice mail, databases, and telephony equipment 

A VA application is created, modified, debugged and tested using the VA 
5 Studio. The completed application is then automatically installed and configured to 
run on the VA Server, which enables tiie VA application to take incoming calls and 
provide access to both public and private information. 

Platform Overview 



An Introduction to Virtual Assistant Applications 

10 A VA application allows a user to manage electronic communications and 

access his or her businesses computer resources through a telephone. Using speech 
recognition and text-to-speech technology, the VA communicates with callers in 
spoken English. By calling into the VA on a standard telephone, a user can perform 
functions such as the following: 

IS • Sending and receiving voice mail messages 

• Checking, replying to, and forwarding email messages 

• Looking up phone numbers and addresses in an electronic address book 

• Accessing information in a compaiqr database 

• Accessing information on the World Wide Web 

20 In addition, the VA can perform many of the functions of a personal secietaiy, 

such as the following: 

• Informing the usct via pager when new voice and email messages arrive 

• Filtering incoming voice mail, email, and pages as instructed by the user 

• Automatically dialing phone numbers 

25 In the preferred embodiment, the VA performs the above functions by 

interfacing with a company's Microsoft Exchange server. This application, in effect, 
allows users to use their desktop Oufiook software over the telephone. 

The VA software includes a development platform (tiie SCE) and run-time 
platform (the SDE), which can host a variety of different VA*s. The SDE provides 

30 the core conq)onents necessary for the functionality of a VA: a telephony inter&ce. 
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speech recognition facilities, a texMo-speech engine, interfaces with databases and 
mail servers, and an administrative fiamework in which fhe assistant applications will 
run. The SCE also includes development tools that programmers can use to create 
custom VA applications. 

5 VA Platform Components 

As discussed above, the VA Platform consists of three main components: 

• The Service Deployment Environment (SDE) 

• Virtual Assistant Applications 

• The Service Creation Environment (SCE) 

1 0 The Amotion of each of tiiese components can be understood using a World 

Wide Web analogy. The SDE functions like a web server, providing connections with 
the network and telephone system, controlling the execution of VA applications, and 
providing resources such as text-to-speech and voice recognition engines that will be 
accessed by fhe applications that run on it 

15 The VA applications are analogous to web pages, determining fhe content that 

will be presented and controlling the interactions with fhe user. A VA application 
uses scripting languages such as VBScript, JavaScript, and Perl, so that developers 
can add significant functionality to a VA, such as performing mathematical 
calculations, processing text, and calling ActiveX and COM objects. 

20 Just as Microsoft Front Page and Netscape Composer are used to create web 

pages, the SCE is the development environment used to create fhe VA applications. 
The main component of fhe SCE is the VA Studio application, which is based on the 
Microsoft Visual Studio paradigm and provides a graphical enviroimient with a 
variety of tools that can be used to create, debug, and publish applications tiiat are run 

25 on the SDE. The SCE also includes a set of COM objects that can be used in 

applications to perform functions such as checking email, reading from a database, 
and manipulating sound files. 

T/ie SDE Service Processes 

The Service Deployment Environment consists of eight processes that run 
30 simultaneously and perform the functions necessary to support a VA application. In 
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the preferred embodiment, each of these SDE components runs as a Windows NT 
Service or background process. 

Although they may all run on fte same hardware platform^ for large VA 
implementations the components can be distributed across several servers and 
S communicate over the network. Such distribution can allow, for example, one server 
to be dedicated to performing voice recognition functions while another supports the 
VA Engine fliat actually runs the applications. When multiple VA components are 
distributed across multiple machines, these machines are collectively termed a VA 
server set 

10 The VA Engine 

As illustrated in Figure 4, the VA Engine 100 is flie virtual machine on which 
a VA application 102 runs. Based on the application's instructions, the VA Engine 
uses its telephony interface 104 to communicate with the user 106 and its speech 
interface 110 to recognize speech into text and translate text into speech. The VA 
15 Engine connects to an Active Scripting Engine 1 12 to execute tiie scripts contained in 
the VA application, and it also communicates with administrative processes such as 
the VA Server 1 14 and VA Manager 1 16. 

In the preferred embodiment, the user is electronically coupled to the virtual 
assistant application via a public switched telephone network. As can be appreciated 
20 by one skilled in the art, the communications network that electronically couples the 
user interface to the computer on which a virtual assistant application is running could 
be a packet switched telephone network. Also, the communications network could be 
a wireless commimications network. 

A VA Engine process can support user interaction over only one telephone 
25 line, but multiple VA Engines can be run simultaneously on a single platform. If flie 
VA platform is connected to more than one telephone line, then a separate VA Engine 
will be running for each incoming line. 

The Text-to-speech (TTS) Server 

The Text-to-Speech Server 120 receives text fiom other components, 
30 translates it into speech (that is, into a sound file), and returns it to the requesting 
component Hiis speech translation service is isolated in a separate component to 
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What is claimed is: 

1 . A computer-based virtual assistant tiie behavior of which can be 
changed by the user, comprising: 

a voice user interface for inputting information into and receiving information 
5 from the virtual assistant by speech; 

a coromunications network; 

a virtual assistant application running on a remote computer, the remote 
computer being electronically coupled to the user inter&ce via the 
communications network; 
10 wherein the behavior of the virtual assistant changes responsive to user input 



2. The virtual assistant of claim 1, wherern the user input is a user 
preference. 

IS 3. The virtual assistant of claim 2, wherein the user preference is a tempo 

setting that controls flie verbosity of flie virtual assistant 

4. The virtual assistant of claim 2, wherein tibe user preference is an 
assertiveness setting that controls whether confirmation fiom the user is required 

20 before the virtual assistant performs a task requested by the user. 

5. The virtual assistant of claim 2, wherein the user preference is 
assertiveness setting that controls whether confirmation firom the user is required 
before the virtual assistant pexforms a task fiiat the virtual assistant anticq}ates that the 
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user will request 

6. The virtual assistant of claim 2, wherein the user preference is a user 
competence level. 

5 

7. The virtual assistant of claim 6, wherein the user competence level is 
selected from the group consisting of novice, experienced and expert 

8. Hie virtual assistant of claim 7, wherein the tempo setting changes 
10 automatically responsive to a change in the user competence level. 

9. The virtual assistant of claim 7, wherein the assertiveness setting 
changes automatically responsive to a change in the user conq)etence level. 

15 10. The virtual assistant of claim 2, wherein the user preference is a detail 

setting that controls the amount of detail that the virtual assistant provides the user 
about an item of information. 

11. The virtual assistant of claim 2, wherein the user preference is a 

20 notification preference setting that controls fte method by which the virtual assistant 
notifies the user of predetermined events. 

12. The virtual assistant of claim 11, wherem the method by which the 
virtual assistant notifies flie user of a predetermined event is selected fix)m the group 
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consisting of a telephone call, a pager notification, an instant messaging service, a 
short messaging service, and an email. 



13. The virtual assistant of claim 1 1; wherein the predetermined event of 
S which the user is notified is selected from tiie group consisting of a reminder for a 
task, a reminder for an appointment, a reminder for an even^ receipt of an email 
message and receipt of a voice mail message. 



14. The virtual assistant of claim 2, wherein the user preference is a 
10 telephone call routing setting that controls the routing of a telephone call to the user. 



15. The virtual assistant of claim 14, wherein the telephone call routing 
setting can be configured to route a telephone call to the user to a predetermined 
telephone number. 

15 

16. The virtual assistant of claim 14, wherein the telephone call routing 
settmg can be configured to route a telephone call to the user to a plurality of 
predetermined numbers, the predetermined number being determined by a 
predetennined schedule for the user. 

20 

17. The virtual assistant of claim 14, wherein the telephone call routing 
setting can be configured not to disturb the us^ by not routing telephone calls to the 
user. 



113 



wo 01/75555 PCTAJSOl/06883 
1 8. The virtual assistant of claim 2, wherein the user preference is a page 
routing setting that controls flie routing of a page to the user. 



1 9. The virtual assistant of claim 1 8, wherein the page routmg setting can 
5 he configured to route a page to the user to a predetermined email address. 

20. The virtual assistant of claim 18, wherein Hie page routing setting can 
he configured to route a page to the user to a plurality of predetermined email 
addresses, the predetermined email addresses being detemiined by a predetermined 

10 schedule for the user. 

21 . The virtual assistant of claim 1 8, wherein the page routing setting can 
be configured not to disturb the user by not routing a page to the user. 

15 22. The virtual assistant of claim 2, wherein the user preference is a 

greeting settmg. 

23. The virtual assistant of claim 22, wherein the greeting settmg can be 
configured so that the virtual assistant communicates to the user information about tiie 

20 number of new messages received by the user. 

24. The virtual assistant of claim 19, wherein the new messages received 
by the user are selected &om the group consisting of voice mail messages, email 
messages, task requests and meeting requests. 
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25. The virtual assistant of claim 22, wherein the greeting setting can be 
configured so that the virtual assistant communicates to the user information about the 
user's appointments ftat are scheduled for a predetermined period of time. 

5 

26. The virtual assistant of claim 22, wherein the greeting setting can be 
configured so that the virtual assistant communicates to the user information about the 
user's tasks that are due during a predetermined period of time. 

10 27. The virtual assistant of claim 2, wherein flie user preference is an 

archive message setting fliat can be configured by the user to specify the location for 
archiving messages. 

28. The virtual assistant of claim 2, wherein the user preference is an 

1 5 operator setting that can be configured by the user to specify a predeteimined niunber 
to which a caller can be routed. 

29. The virtual assistant of claim 2, wherein the user preference is a tips 
setting that can be configured by the user to enable or disable the providing of hints to 

20 theusCT. 

30. The virtual assistant of claim 2, wherein flie user preference is a 
politeness setting that, when enabled, causes the virtual assistant to include words or 
phrases associated with polite discourse in the output from the virtual assistant 
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3 1 . The virtual assistant of claim 30, wherein the words or phrases 
associated with polite discourse are selected from the ffoxxp consisting of **please," 
"thank you," *thanks " "excuse " •*pardon," "may I," and "would you mind." 

5 



32. The virtual assistant of claim 1 , wherein the cormnunications network 
is a public switched telephone netwoik. 

10 33. The virtual assistant of claim 1, wherein the communications netwodc 

is a packet switched telephone netwoik. 

34. The virtual assistant of claim 1, wherein the communications netwoik 
is a wireless telecommunications network. 

15 

35. A computer-based virtual assistant that automatically adapts its 
behavior^ comprising: 

a voice user interface for iiq)utting information into and receiving information 

from the virtual assistant by speech; 
20 a communications network; 

a virtual assistant application running on a remote computer, the remote 

computer being electronically coupled to the user inter&ce via flie 

communications network; 

wherein the remote computer is programmed to automatically change the 

25 behavior of the virtual assistant responsive to input recdved by the 
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virtual assistant 

36. The virtual assistant of claim 35, wherein the input received by the 
virtual assistant is comprised of information about the user. 

5 

37. The virtual assistant of claim 36, wherein the user infoimation is 
comprised of information about the user's experience wifh the virtual assistant 

38. The virtual assistant of claim 37, wherein the user experience 

1 0 information is comprised of the number of times that the user has accessed the virtual 
assistant 

39. The virtual assistant of claim 38, wherein tips are automatically 
disabled after the user accesses the virtual assistant a predetermined number of times. 

15 

40. The virtual assistant of claim 39, wherem the predetermmed number of 
times is a multiple of the number of tips. 

41 . The virtual assistant of claim 40, wherein the multiple is between two 
20 and t^ 

42. The virtual assistant of claim 4 1 , wherein the multiple i& three. 

43. The virtual assistant of claim 38, wherein the message of the day is 
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automatically disabled after the user accesses the virtual assistant a predetermined 
number of times. 



44. The virtual assistant of claim 43» wherein the predetermined number of 
S times is a multiple of the number of tips. 

45. The virtual assistant of claim 44, wherein the multiple is between two 
and ten. 

10 46. The virtual assistant of claim 4S, wherein the multiple is three. 

47. The virtual assistant of claim 3 8, wherein the virtual assistant 
automatically provides a prompt to ihe user in response to which the user can increase 
the tempo of the virtual assistant, the prompt being provided automatically after the 

1 5 user accesses the virtual assistant a predetermined number of times. 

48. The virtual assistant of claim 47, wherein the predetermined number is 
more than one. 

20 49. The virtual assistant of claim 48, wherein the predetermined number is 

between ten to thirty. 

SO. The virtual assistant of claim 49, wherein the predetermined number is 

twenty. 
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51. The virtual assistant of claim 38» wheiein the virtual assistant 
automatically increases the tempo of the virtual assistant after the user has accessed 
the virtual assistant a predetermined number of times. 

5 

52. The virtual assistant of claim 5 1 , wherein the predetermined number of 
times is more than one. 

53. The virtual assistant of claim 52, wherein the predetermined number is 
10 between ten to thirty. 

54. The virtual assistant of claim 53, wherein the predetermined number is 

twenty, 

1 5 55. The virtual assistant of claim 38, wherein the virtual assistant 

automatically provides a prompt to the user in response to which the user can increase 
the assertiveness of the virtual assistant, the prompt being provided automatically 
after the user accesses the virtual assistant a predetermined number of times. 

20 56. The virtual assistant of claim 55, wherein &e predetennined numbo: is 

more &an one. 

57. The virtual assistant of claim 56, wherein the predetennined number is 
between ten to thirty. 
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58. The virtual assistant of claim 57, wherein fhe predetermined number is 

twentyr 

5 59. The virtual assistant of claim 38, wherein flie virtual assistant 

automatically increases the assertiveness of the virtual assistant after the user has 
accessed the virtual assistant a predetemiined number of times. 

60. Hie virtual assistant of claim 59, wherein the predetermined number of 
10 times is more than one. 

61 • The virtual assistant of claim 60, wherein the predetermined number is 
between ten to fhirfy. 

1 5 62. Hie virtual assistant of claim 61 , wherein the predetermined number is 

twenty. 

63. The virtual assistant of claim 37, wherein the user e)q>erience 
information is conq)rised of the number of times per day that the user has accessed the 

20 virtual assistant 

64. The virtual assistant of claim 63, wherein tips are automatically 
disabled after the user accesses the virtual assistant a predetermined number of times 
per day. 
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65. The virtual assistant of claim 64, wherein the predetermined number is 
one or more. 

5 66. The virtual assistant of claim 63, wherein the message of the day is 

automatically disabled after the user accesses the virtual assistant a predetenmined 
number of tunes per day. 

67. The virtual assistant of claim 66, wherein the predetermined number is 
10 one or more. 

68. The virtual assistant of claim 37, wherein the user experience 
information is comprised of the amount of time between user sessions. 

1 5 69. The virtual assistant of claim 68, wherein the tips are automatically 

disabled during the current user session if the time since ^t last user session is a 
predetermined amount of time. 

70. The virtual assistant of claim 69, wherein the predetermined amount of 
20 time is ten minutes or less. 

71 . The virtual assistant of claim 68, wherein the message of the day is 
automatically disabled during flie user session if the time since the last user session is 
a predetermined amount of time. 
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72. The virtual assistant of claim 71, wherein the predetermined amount of 
time is ten minutes or less. 

5 . 73. The virtual assistant of claim 68, wherein time-of-day specific 

greetings are automatically disabled during the current user session if the time since 
the last user session is a predetermined amount of tmie. 

.74. The virtual assistant of claim 73, wherein the predeteimined amount of 
10 time is ten minutes or less. 

75. The virtual assistant of claim 68, wherein an appointments setting is 
automatically disabled during the current user session if tiie time since tbe last user 
session is a first predetermined amount of time. 

15 

76. The virtual assistant of claim 75, wherem the first predetermined 
amount of time is ten minutes or less. 

77. The virtual assistant of claim 75, wherein the appointments setting, 
20 when enabled, causes the virtual assistant provide the user information about the 

user's appointments for a second predetermined amount of time. 

78. The vutual assistant of claim 77, wherein tiie second predetermined 
amount of time is one day. 
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79. The virtual assistant of claim 68, wherein a tasks setting is 
automatically disabled during the cunent user session if the time since &e last user 
session is a first predetermined amount of time. 

80. The virtual assistant of claim 79, wherein tiie first predetermined 
amount of time is ten minutes or less. 



81 • The virtual assistant of claim 79, wherein the tasks setting, when 
10 enabled, causes the virtual assistant provide the user information about the user's 
tasks that are due for a second predetermined amount of time. 

82. The virtual assistant of claim 81, wherein flie second predetermined 
amount of tune is one day. 

15 

83. The virtual assistant of claim 37, wherein the user experience 
information is comprised of an amount of tune a user pauses during the recording of a 
message. 



20 84. The virtual assistant of claim 83, wherein the amount of time a user 

pauses during the recording of a message is the average amount of time a user pauses 
during the recording of a message. 

85. The virtual assistant of claim 83, wherem, if the amount of time a user 
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pauses is greater Hbm a first predetermined amount of time, the virtual assistant stops 
recording, provides the user with the option to continue recording and the first 
predetermined amount of time is automatically increased by a second predetermined 
amount of time. 

5 

86. The virtual assistant of claim 85, wherein the first predetermined 
amount of time is two seconds. 



87. The virtual assistant of claim 85, wherein the second predetermined 
10 amount of time is 500 milliseconds. 

88. The virtual assistant of claim 37, wherem the user experience 
information is comprised of whetiier words associated with polite discourse are 
included in input received fix>m ihe user. 

15 

89. The virtual assistant of claim 88, wheiem the words associated with 
polite discourse are selected firom the group consisting of *'please," ••thank you," 
••thanks," •'excuse," ••pardon," "may I," and • Vould you mini" 

20 90. Hie virtual assistant of claim 88, wheiem a politeness setting is 

automatically enabled when the user input includes words associated with polite 
discourse. 



9 1 • ITie virtual assistant of claim 90, wherein, if the politeness setting is 
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enabled, the output of the virtual assistant includes words associated with polite 
discourse. 



92. The virtual assistant of claim 91, wherein &e words associated with 
5 pleasant discourse are selected from the group consisting of **please," "thank you,'* 
**thanks," "excuse," "pardon," "may I," and * Vould you mind." 



93, The virtual assistant of claim 36, wherein tiiie user information is 
comprised of information about the user's emotion. 

10 

94. The virtual assistant of claim 93, wherein the information about the 
user's emotion is selected from the group consisting of voice volume, word choice 
and ^eech rate. 

15 95. The virtual assistant of claim 93, wherein a user's emotional state is 

determined by the virtual assistant based on the information about the user's emotion. 

96. The virtual assistant of claim 95, wherein the user's emotional state is 
selected from the group consisting of calm and angiy. 

20 

97. The virtual assistant of claim 96, wherein, if the user's emotional state 
is angry, the output of the virtual assistant automatically includes words associated 
with submissive discourse. 
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98. The virtual assistant of claim 99, wherein fhe words associated with 
submissive discourse include "sorry," "regref* and "apologize." 



99. The virtual assistant of claim 93, wherein the user emotion information 
5 is obtained firom a current user session. 



100. The virtual assistant of claim 99, wherein flie user emotion information 
obtained during &e current user session is used in future user sessions. 



10 101 . The virtual assistant of claim 36, wherein fhe user information is 

comprised of information about the amount of time since the user last provided ixxpat 
to fhe virtual assistant 



102. The virtual assistant of claim 101, wherein, if fhe amount of time since 
15 the user last provided input to fhe virtual assistant is a predetermined amount of time, 
fhe virtual assistant automatically asks fhe user if fhe user wants to perform a 
predetermined action. 



103. The virtual assistant of claim 102, wherein, if fhe amount of time since 
20 the user last provided input to the virtual assistant is a predetermined amount of time, 
the virtual assistant automatically performs a piedeteimined action. 



104. The virtual assistant of claim 103, wherein, if the amount of time since 
fhe user last provided input to the virtual assistant is a predetermined amount of time, 
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. the virtual assistant automatically provides the user a predetermined Unt as to an 
action that can be performed by the. virtual assistant 



105. The virtual assistant of claim 103, wherein the predetermined amount 
5 oftime is fifteen seconds or more. 

1 06. The virtual assistant of claim 35, wherein the communications network 
is a public switched telephone netwodc 

10 107. The virtual assistant of claim 35, wherein tiie communications network 

is a packet switched telephone network. 

108. The virtual assistant of claim 35, wherein the communications netwoik 
is a wizieless telecommunications netwoik. 

15 

109. A computer-based virtual assistant the behavior of which can be 
changed by the user, comprising: 

a user interface for inputting information into and receiving information from 

the virtual assistant; 
20 a coxmnunications network; 

a virtual assistant application ruiming on a remote computer, the remote 

computer being electronically coupled to the user interface via the 

conmiunications networlq 
wherein the behavior of the virtual assistant changes responsive to user input 
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1 10. Hie virtual assistant of claim 109, wherein the user interface is a 
telephone and the user inputs infoimation into and receives information from the 
virtual assistant via speech. 

5 

111. The virtual assistant of claim 1 09, wherein the user interface is a 
telephone and the user inputs information into the virtual assistant via DTMF tones 
and receives information from the virtual assistant via speech. 

10 1 12. The virtual assistant of claim 109, wherein the user interface is a 

graphical user inter&ce. 

113. The virtual assistant of claim 109, wherein the user interface is a 
personal digital assistant 

IS 

1 14. The virtual assistant of claim 109, wherein the conmiunications 
network is a public switched telephone network. 

115. Ihe virtual assistant of claim 109, wherein the communications 
20 network is a packet switched telq>hone network. 

116. The virtual assistant of claim 109, wherein the communications 
network is a wireless telecommunications netwodc. 
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1 17. A computer-based virtual assistant that automatically adapts its 
behavior, comprising: 

a voice user interface for inputting information into and receiving information 
from the virtual assistant by speech; 
5 a communications netwoilq 

a virtual assistant application running on a remote computer, the remote 
computer being electronically coupled to the user inter&ce via the 
communications network; 
wherein the remote computer is programmed to automatically change the 
10 behavior of flie virtual assistant responsive to input received by the 

virtual assistant 

118. The virtual assistant of claim 1 17, wherein the user interface is a 
telephone and the user inputs information into and receives information from the 

15 virtual assistant via speech. 



119. The virtual assistant of claim 1 17, wherein the user inter&ce is a 
telephone and the user inputs information into tiie virtual assistant via DTMF tones 
and receives information from the virtual assistant via speech. 

20 

120. The virtual assistant of claim 117, wherein the user int^&ce is a 
graphical user interface, 

121. The vutual assistant of claim 1 17, wherein the user interface is a 
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personal digital assistant 

122. The virtual assistant of claim 1 17, whrarein the communications 
network is a public switched telephone networL 

5 

123. The virtual assistant of claim 1 17, wherein the communications 
network is a packet switched telephone network. 

124. The virtual assistant of claim 1 17, wherein the communications 
1 0 network is a wireless teleccxmmunications network. 

125. A computer based method for performing a first command via a voice 
user inter&ce on information stored in a computer memory, wherein the information 
upon which the first command is to be performed is comprised of a subset of objects 

15 specified by the usct, the subset being selected fi:om a set of objects, wherein each 
object in the set of objects has an object type, at least one taggable field associated 
with the object type, and a corresponding value for the at least one taggable field, ttie 
method comprising the steps of. 

storing the set of objects in the computer memory; 
20 receiving firom the user via the voice user inter&ce an utt»:ance, the utterance 

being comprised of a first command, an object type selection, a 
taggable field selection, and a value for the selected taggable field; 
responsive to the utterance, retrieving a subset of objects firom the set of 

objects, the subset containing at least one object, the at least one object 
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being of fhe object type selected by the user^ having a value in the 
taggable field selection that matches the taggable field value received 
firom the user, and 
perfonning ttie first command on the retrieved subset of objects. 

5 

126. The method of claim 125, wherein the object type is selected from the 
group consisting of voicemail message, email message, meeting request, task request 
and&x. 



10 127. The method of claim 126, wherem the taggable field for the voicemail 

message type is selected fix>m the group consisting of caller's name, callback number, 

delivery date, subject and status. 
« 

128. The method of claim 127, wherein the taggable field value for the 

15 status field is selected fi:om the group consisting of first, last, new, old, read, unread, 
deleted and urgent 

129. The method of claim 126, wherein the taggable field for the email 
message type is selected fi:om the group consisting of sender, delivery date, subject 

20 and status. 



130. The method of claim 129, wherein the taggable field value for the 
status field is selected firom the group consisting of first, last, new, old, read, unread, 
deleted and urgent 
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13 1 . The me&od of claim 126, wberein fhe taggable field for fte meeting 
request message type is selected from the group consisting of sender, delivery date, 
subject, location, start time, end time, and status. 

5 

132. The method of claim 131, wherein the taggable field value for the 
status field is selected horn the group consisting of first, last, new, old, read, unread, 
deleted and urgent 

10 133. The method of claim 126, wherein fhe taggable field for fhe task 

request message type is selected from fhe group consisting of sender, delivery date, 
subject, due date, owner, status and percentage complete. 

134. The method of claun 133, wherein the taggable field vahie for fhe 

IS status field is selected from fhe group consisting of first, last, new, old, read, unread, 
deleted and uigrat 

135. The method of claim 126, wherein fhe taggable field for the fiax 
message type is selected from the group consisting of telephone number of the 

20 machine that sent fhe fax, sender, delivery date, subject and status. 

136. The method of claim 135, wherein the taggable field value for the 
status field is selected fix>m fhe group consisting of first, las^ new, old, read, unread, 
deleted and urgent 

132 



wo 01/75555 



PCTAJSOl/06883 



137. The meHiod of claim 125, wherein ihe first command is selected fixnn 
the group consisting of count, browse, list and read. 

5 138. The method of claim 125, wherein the object type is a contact 

139. The method of claim 138, wherein the taggable field is selected fixmi 
the group consisting of first name, last name and nickname. 

10 140. The mettiod of claim 138, wherein ttie first command is selected fixnn 

flie group consisting of comit, browse, list and read. 

141 . The method of claim 125, further including tide step of: 
receiving a second command firom the user, the second command to be 
1 5 performed on an active object in the subset of objects, the at least one 

active object being the object upon which the first command was 
performed ; and 
performing the second command on &e active object 

20 142. The method of claim 141, wh^ein the second command is selected 

firom the group consisting of next, previous, first, last, read, get more detail, file, 
delete and restore. 



143. A computer readable medium containing instructions for controlling a 
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computer system to perfonn a first coimnand via a voice user interface on information 
stored in a computer memory, wherein the information upon which the first command 
is to be performed is comprised of a subset of objects specified by &e user, the subset 
being selected &om a set of objects, wherein each object in the set of objects has an 
S object type, at least one taggable field associated with the object type, and a 
corresponding value for the at least one taggable field, by: 
storing the set of objects in &e computer memory; 

receiving firom the user via the voice user interface an utterance, the utterance 
being comprised of a first command, an object type selection, a 
10 taggable field selection, and a value for the selected taggable field; 

responsive to the utterance, retrieving a subset of objects firom the set of 

objects, the subset containing at least one object, the at least one object 
being of the object type selected by die user, having a value in the 
taggable field selection that matches the taggable field value received 
1 5 &om the user, and 

performing the first command on the retrieved subset of objects. 



1 44. A computer readable medium containing instructions for controlling a 
computer system to perfonn a first command via a voice user interface on information 
20 stored in a computer memory, wherein the information upon which the first command 
is to be performed is comprised of a subset of objects specified by the user, the subset 
bemg selected firom a set of objects, wherein each object in the set of objects has an 
object type, at least one taggable field associated with ttie object type, and a 
corresponding value for the at least one taggable field, by: 
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Storing the set of objects in the computer memoiy; 

receiving from &e user via the voice visesi interface an utterance, Oxt utterance 
being comprised of a first conunand» an object type selection, a 
taggable field selection, and a value for the selected taggable field; 
5 responsive to the utterance, retrieving a subset of objects from the set of 

objects, the subset containing at least one object, the at least one object 
being of the object type selected by the user, having a value in the 
taggable field selection Aat matches the taggable field value received 
from the user, 

10 performing the first command on the retrieved subset of objects; 

receiving a second command from the user, the second command to be 

performed on an active object in the subset of objects, the at least one 
active object being the object upon which flie first command was 
p^ormed; and 

15 performing the second command on the active object 

145. A method for receiving from a virtual assistant application user a 
desired choice from a plurality of choices presented to the user by the virtual assistant, 
comprised of: 

20 providing a choice stream to the user, the choice steam being comprised of at 

least one choice prompt, the choice prompt being associated wifli a 
choice available to the user and the choice prompt bemg comprised of 
a recorded utterance, and a corresponding active segment, wherein the 
user can select the choice associated with the choice prompt during the 
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conesponding active segment; 
receiving input from the user during the active segment corresponding to the 
desired choice prompt, flie input that indicating that the user desires to 
select the choice associated with the choice prompt, and 
S performing an action associated with the choice selected by the user. 



146. The method of claim 145, further comprising the following step prior 
to the step of providing a choice stream: 

receiving a predefined input from the user indicating ibsA the user desires to 
10 enter a choice prompt mode. 



147. The method of claim 146, further comprising the step of: 
receiving the predefined input from the user indicating that the user desires to 

exit the choice prompt mode. 

15 

148. The method of claim 145, further comprising the following step prior 
to ttie step of providing tiie choice stream: 

providing a predefined audible signal to the user, the audible signal signifying 
to the user the beginning of a choice stream. 

20 

149. The method of claim 145, wherein the length of the active segment is 
not less ttian 200 milliseconds. 

150. The mefliod of claim 145, wherein first choice prompt in flie choice 



136 



wo 01/75555 PCTAJSOl/06883 
Stream is fhe choice automatically detennined by the virtual assistant to be the choice 
likely to be selected by Hie user. 



151. The method of claim 1 50, wherein fhe choice likely to be selected by 
5 the user is automatically detennined by the virtual assistant based on the user's past 
choices. 



1 52. The method of claim 145, further comprising fhe followmg step prior 
to the step of perfoiming fhe action: 
10 repeating fhe choices in fhe choice stream until the user selects one of the 

choices in the choice stream. 



1 53. The metiiod of claim 145, wherein fhe input received fiom fhe user 
during fhe active segment is predefined verbal input 

15 

154. The method of claim 153, wherein fhe predefined verbal input is 
selected from fhe group of words consisting of 'Ves" and '^o.** 

155. The method of claim 145, wherein fhe input received firom fhe user 
20 during the active segment is predefined nonveibal ioput 

156. The method of claiml 55, wherein fhe predefined nonverbal input is a 
predefined DTMF. 
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157. The mefliod of claim 156, wh^ein the predetennined DTMF is flie 
DTMF generated by a telephone keypad wifh fhe star key is pressed. 



158. A method for receiving from a virtual assistant application user a 

5 desired choice from a plurality of choices presented to ihe user by &e virtual assistant 
comprised of. 

receiving a predefined input from the user indicating that the user desires to 
enter a choice prompt mode; 

providing a choice stream to the user, the choice steam being comprised of at 
10 least one choice prompt, fhe choice prompt being associated wifh a 

choice available to fhe user and the choice prompt being comprised of 
a recorded utterance, and a corresponding active segment, vi^herein the 
user can select fhe choice associated with fhe choice pronq)t during the 
corresponding active segment 
1 5 receiving the predefined input from tite user during the active segment 

corresponding to fhe desired choice prompt, fhe predefined input that 
indicating that fhe user desires to select fhe choice associated wifh fhe 
choice prompt; 

performing an action associated with the choice selected by the user, and 
20 receiving fhe predefined input &om die user indicating that fke user desires to 

exit fhe choice prompt mode. 

159. The method of claim 1 58, fiirther comprising the following step prior 
to fhe stsp of providing the choice stream: 
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providing a predefined audible signal to the user, the audible signal signifying 
to the user the beginning of a choice stream. 



160. The method of claim 145, wherein the length of the active segment is 
5 not less than 200 milliseconds. 

161. Hie method of claim 158, wherem first choice prompt in the choice 
stream is the choice automatically determined by the virtual assistant to be the choice 
likely to be selected by the user. 

10 

162. The me&od of claim 161, wherein the choice likely to be selected by 
the user is automatically determined by the virtual assistant based on the user*s past 
choices. 

15 163. The method of claim 158, wherein the input received fi:om the user 

during the active segment is predefined verbal input 

1 64. The method of claim 1 63, wherein the predefined verbal input is 
selected firom the group of words consisting of **y^s" and "no." 

20 

1 65. The method of claim 1 58, wherein the input received £tom the user 
during the active segment is predefined nonv^bal input 



1 66. The method of claim 1 65, wherein the predefined nonveibal input is a 

139 



wo 01/75555 PCTAJSOl/06883 
predefined DTMF. 

167. The method of claim 166, wherein flie predetermined DTMF is flie 
DTMF generated by a telephone keypad with the star is pressed, 

5 
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Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



